IN-DEPTH ANALYSIS

RoTBench: A Multi-Level Benchmark for Evaluating the Robustness of Large Language Models in Tool Learning (2024.emnlp-main)

Challenge:	Existing work on LLMs that only enhance reasoning abilities, but which lack factual hallucination and slow-thinking capabilities, argues that SPP is a cognitive synergist.
Approach:	They propose a Solo Performance Prompting (SPP) that transforms a single LLM into a cognitive synergist by engaging in multi-turn self-collaboration with multiple personas.
Outcome:	The proposed model reduces factual hallucination and maintains strong reasoning abilities on three challenging tasks .

Copied to clipboard

Junjie Ye, Yilong Wu, Songyang Gao, Caishuang Huang, Sixian Li, Guanyu Li, Xiaoran Fan, Qi Zhang, Tao Gui, Xuanjing Huang

RethinkingTMSC: An Empirical Study for Target-Oriented Multimodal Sentiment Classification (2023.findings-emnlp)

Challenge:	Current research emphasizes LLMs’ capacity to utilize tools in well-structured environments while overlooking their stability when confronted with the inevitable noise of the real world.
Approach:	They propose a multi-level benchmark to evaluate the robustness of large language models in tool learning by establishing five external environments with varying levels of noise.
Outcome:	The proposed model outperforms the GPT-4 model in tool learning in three critical phases: tool selection, parameter identification, and content filling.

Copied to clipboard

Junjie Ye, Jie Zhou, Junfeng Tian, Rui Wang, Qi Zhang, Tao Gui, Xuanjing Huang

Does Character-level Information Always Improve DRS-based Semantic Parsing? (2023.starsem-1)

Challenge:	Recent studies have shown that current TMSC systems rely on textual information, and the progress in tackling this task has slowed down.
Approach:	They propose to integrate both visual and textual information to improve the performance of TMSC by considering multimodal information.
Outcome:	The proposed model integrates both visual and textual information to improve performance.

Copied to clipboard

Tomoya Kurosawa, Hitomi Yanaka

I Know, but I Don’t Know! How Persona Conflict Undermines Instruction Adherence in Large Language Models (2026.findings-eacl)

Challenge:	incorporating character-level information does not improve the performance in English and German, and is not sensitive to correct character order in Dutch.
Approach:	They propose to incorporate character-level representations into a neural semantic parser for Discourse Representation Structures and to test their performance using order of character sequences.
Outcome:	The proposed parser improves in English, German, Dutch, and Italian in four languages.

Copied to clipboard

Seonmin Koo, Jinsung Kim, Heuiseok Lim

Dialogue Systems & Conversational Agents

Challenge:	Existing studies on persona-grounded dialogue assume idealized scenarios where persona and user utterances are fully aligned.
Approach:	They propose a taxonomy that categorizes model behaviors into three response types . they propose sycophantic, adherent, and wavering responses as response types.
Outcome:	The proposed framework categorizes model behaviors into three response types and develops a measurement schema grounded in this taxonomy.

A Perspective on LLM Data Generation with Few-shot Examples: from Intent to Kubernetes Manifest (2025.acl-industry)

Copied to clipboard

Antonino Angi, Liubov Nedoshivina, Alessio Sacco, Stefano Braghin, Mark Purcell

LED: A Dataset for Life Event Extraction from Dialogs (2023.findings-eacl)

Challenge:	Large Language Models (LLMs) have transformed how complex tasks can be automated . traditional cloud computing operations involve complex manual configurations .
Approach:	They propose a pipeline for generating K8s manifests directly from user-described intents expressed in natural language using LLMs.
Outcome:	The proposed pipeline can generate K8s manifests directly from user-described intents expressed in natural language using LLMs.

Copied to clipboard

Yi-Pei Chen, An-Zi Yen, Hen-Hsen Huang, Hideki Nakayama, Hsin-Hsi Chen

Information Extraction & Text Mining

Challenge:	Existing methods for extracting life events from conversations are limited.
Approach:	They propose a dataset containing fine-grained life event annotations on conversational data.
Outcome:	The proposed dataset combines three information extraction frameworks to extract life events from conversations.

Moleco: Molecular Contrastive Learning with Chemical Language Models for Molecular Property Prediction (2024.emnlp-industry)

Copied to clipboard

Jun-Hyung Park, Hyuntae Park, Yeachan Kim, Woosang Lim, SangKeun Lee

Understanding the Effects of Noise in Text-to-SQL: An Examination of the BIRD-Bench Benchmark (2024.acl-short)

Challenge:	Pre-trained chemical language models (CLMs) are highly effective in predicting molecular properties, but implicitly contain limited structural information.
Approach:	They propose a contrastive learning framework to enhance the understanding of molecular structures within chemical language models by measuring the similarity of fingerprint vectors among different molecules.
Outcome:	The proposed framework outperforms state-of-the-art models in molecular property prediction.

Copied to clipboard

Niklas Wretblad, Fredrik Riseby, Rahul Biswas, Amin Ahmadi, Oskar Holmström

Programming Languages in NLP

Challenge:	BIRD-Bench was created to model dirty and noisy database values, but it was not created to contain noise and errors in the questions and gold SQL queries.
Approach:	They analysed the distribution and types of noise in the widely used BIRD-Bench benchmark and the impact of noise on models.
Outcome:	The BIRD-Bench benchmark contains ambiguous questions and syntactical errors, but it was not designed to contain noise and errors in the questions and gold SQL queries.

Dialz: A Python Toolkit for Steering Vectors (2025.acl-demo)

Copied to clipboard

Zara Siddique, Liam Turner, Luis Espinosa-Anke

Challenge:	Dialz is a Python library for advancing research on steering vectors for open-source LMs.
Approach:	They propose a Python library for advancing research on steering vectors for open-source LMs.
Outcome:	The proposed method reduces harmful outputs and provides insights into model behaviour across different layers.

Do Vision-Language Models Understand Compound Nouns? (2024.naacl-short)

Copied to clipboard

Sonal Kumar, Sreyan Ghosh, S Sakshi, Utkarsh Tyagi, Dinesh Manocha

A Process-oriented Dataset of Revisions during Writing (2020.lrec-1)

Challenge:	Open-vocabulary vision-language models (CLIP) are emerging as a promising new paradigm for text-to-image retrieval.
Approach:	They propose a benchmark to evaluate the effectiveness of open-vocabulary vision-language models (CLIP) for text-to-image retrieval using contrastive loss.
Outcome:	The proposed framework improves CN understanding of CLIP by 8.25% on Compun.

Copied to clipboard

Rianne Conijn, Emily Dux Speltz, Menno van Zaanen, Luuk Van Waes, Evgeny Chukharev-Hudilainen

Challenge:	Revisions are defined as "changes at any point in the writing process" a dataset of 7,120 revisions was created to analyze revisions in writing .
Approach:	They use keystroke data and eye tracking data of 65 students to analyze revisions . they define revisions as "changes at any point in the writing process"
Outcome:	The proposed dataset includes 7,120 revisions from 65 students from different backgrounds . each type of revision can have a different effect on the written product or writing quality .

FinDABench: Benchmarking Financial Data Analysis Ability of Large Language Models (2025.coling-main)

Copied to clipboard

Shu Liu, Shangqing Zhao, Chenghao Jia, Xinlin Zhuang, Zhaoguang Long, Jie Zhou, Aimin Zhou, Man Lan, Yang Chong

A Comparison of Transformer and Recurrent Neural Networks on Multilingual Neural Machine Translation (C18-1)

Challenge:	Large Language Models (LLMs) have demonstrated impressive capabilities across a wide range of tasks, but their proficiency and reliability in the specialized domain of financial data analysis remain uncertain.
Approach:	FinDABench is a benchmark designed to evaluate the financial data analysis capabilities of Large Language Models (LLMs) it comprises 15,200 training instances and 8,900 test instances, all meticulously crafted by human experts.
Outcome:	FinDABench measures the financial data analysis capabilities of large language models (LLMs) across three dimensions: 1) Core Ability; 2) Analytical Ability; 3) Technical Ability.

Copied to clipboard

Surafel Melaku Lakew, Mauro Cettolo, Marcello Federico

Reformulating Unsupervised Style Transfer as Paraphrase Generation (2020.emnlp-main)

Challenge:	Recent studies have shown that multilingual NMT models can handle more than one translation direction with a single system.
Approach:	They propose a multilingual neural machine translation model that can handle more than one translation direction with a single system.
Outcome:	The proposed model performs well in low-resource settings against bilingual systems.

Copied to clipboard

Kalpesh Krishna, John Wieting, Mohit Iyyer

Text Generation

Challenge:	Existing systems for style transfer warp the input’s meaning through attribute transfer, which changes semantic properties such as sentiment.
Approach:	They propose a method for fine-tuning pretrained language models on automatically generated paraphrase data to improve the efficiency of style transfer.
Outcome:	The proposed method outperforms state-of-the-art style transfer systems on human and automatic evaluations and proposes fixed variants.

Does GPT-3 Generate Empathetic Dialogues? A Novel In-Context Example Selection Method and Automatic Evaluation Metric for Empathetic Dialogue Generation (2022.coling-1)

Copied to clipboard

Young-Jun Lee, Chae-Gyun Lim, Ho-Jin Choi

Dialogue Systems & Conversational Agents

Challenge:	Empathy is a multi-dimensional concept consisting of cognitive and affective aspects.
Approach:	They propose two new in-context example selection methods that utilize emotion and situational information.
Outcome:	The proposed method is effective in measuring the degree of human empathy.

“Will You Find These Shortcuts?” A Protocol for Evaluating the Faithfulness of Input Salience Methods for Text Classification (2022.emnlp-main)

Copied to clipboard

Jasmijn Bastings, Sebastian Ebert, Polina Zablotskaia, Anders Sandholm, Katja Filippova

Do Context-Aware Translation Models Pay the Right Attention? (2021.acl-long)

Challenge:	Existing work on faithfulness evaluation is not conclusive and does not provide a clear answer as to how different methods are to be compared.
Approach:	They propose a protocol for faithfulness evaluation that makes use of partially synthetic data to obtain ground truth for feature importance ranking.
Outcome:	The proposed method is based on partially synthetic data and is compared with lexical shortcuts on a range of datasets and LSTM models.

Copied to clipboard

Kayo Yin, Patrick Fernandes, Danish Pruthi, Aditi Chaudhary, André F. T. Martins, Graham Neubig

What Makes Math Word Problems Challenging for LLMs? (2024.findings-naacl)

Challenge:	Context-aware machine translation models fail to leverage contextual information to resolve ambiguous words and pronouns.
Approach:	They propose a new dataset that includes supporting context words for 14K translations that professional translators found useful for pronoun disambiguation.
Outcome:	The proposed model can automatically disambiguate pronouns and polysemous words when they are not in the same context.

Copied to clipboard

Kv Aditya Srivatsa, Ekaterina Kochmar

Numerical Reasoning

Challenge:	Experiments show that even quite powerful LLMs are still challenged by MWPs.
Approach:	They propose to analyze what makes math word problems (MWPs) in English challenging for large language models (LLMs).
Outcome:	The proposed model can handle a range of core NLP tasks, but it has emergent abilities, such as ability to solve mathematical puzzles.

Open Political Corpora: Structuring, Searching, and Analyzing Political Text Collections with PoliCorp (2025.emnlp-demos)

Copied to clipboard

Nina Smirnova, Muhammad Ahsan Shahid, Philipp Mayr

Unleashing the Power of Discourse-Enhanced Transformers for Propaganda Detection (2024.eacl-long)

Challenge:	PoliCorp provides researchers with access to rich textual data, enabling in-depth analysis of parliamentary discourse over time.
Approach:	They present a web portal that allows researchers to search political text corpora . the platform currently contains a collection of transcripts from the german parliament .
Outcome:	The proposed platform provides researchers with access to rich textual data, enabling in-depth analysis of parliamentary discourse over time.

Copied to clipboard

Alexander Chernyavskiy, Dmitry Ilvovsky, Preslav Nakov

Entity-level Cross-modal Learning Improves Multi-modal Machine Translation (2021.findings-emnlp)

Challenge:	Existing systems focused on the surface words, ignoring the linguistic structure of the texts.
Approach:	They propose to use discourse analysis to analyze paragraph-level and token-level classifications and propose a Transformer architecture that can be used to detect propaganda.
Outcome:	The proposed system improves on English and Russian texts and shows strong correlations between propaganda instances and discourse spans.

Copied to clipboard

Xin Huang, Jiajun Zhang, Chengqing Zong

An Empirical Study on the Robustness of Massively Multilingual Neural Machine Translation (2024.lrec-main)

Challenge:	Multi-modal machine translation aims at improving translation performance by incorporating visual information.
Approach:	They propose an explicit entity-level cross-modal learning approach that aims to augment the entity representation by combining a translation task and a reconstruction task.
Outcome:	The proposed approach achieves comparable or even better performance than state-of-the-art models.

Copied to clipboard

Supryadi Supryadi, Leiyu Pan, Deyi Xiong

Memory-enhanced Large Language Model for Cross-lingual Dependency Parsing via Deep Hierarchical Syntax Understanding (2025.findings-emnlp)

Challenge:	Recent years have witnessed that massively multilingual neural machine translation (MMNMT) achieves a remarkable progress in both high- and low-resource language translation.
Approach:	They propose to use a robustness evaluation benchmark dataset to assess the translation robustness of Indonesian-Chinese translation in the face of various naturally occurring noise.
Outcome:	The proposed dataset is publicly available at https://github.com/ID-ZH-MTRobustEval.

Copied to clipboard

Jianjian Liu, Ying Li, Zhengtao Yu, Shun Su, Shengxiang Gao, Yuxin Huang

Cross-lingual Science Journalism: Select, Simplify and Rewrite Summaries for Non-expert Readers (2023.acl-long)

Challenge:	Experimental results show that our approach can significantly improve the parsing accuracy of all baseline models, leading to new state-of-the-art results.
Approach:	They propose a deep hierarchical syntax understanding approach to improve the cross-lingual semantic memory capability of large language models by implicitly aligning linguistic knowledge between source and target languages.
Outcome:	The proposed approach improves the cross-lingual semantic memory capability of large language models by combining implicit multi-task fine-tuning and explicit label bank guiding.

Copied to clipboard

Mehwish Fatima, Michael Strube

Text Generation

Challenge:	CSJ is a task of text simplification and cross-lingual scientific summarization to facilitate science journalists’ work.
Approach:	They propose to combine CSJ tasks SELECT, SIMPLIFY and REWRITE to produce cross-lingual simplified science summaries for non-expert readers.
Outcome:	The proposed task outperforms existing solutions on Wikipedia and can serve as a strong baseline for future work.

Composable Sparse Fine-Tuning for Cross-Lingual Transfer (2022.acl-long)

Copied to clipboard

Alan Ansell, Edoardo Ponti, Anna Korhonen, Ivan Vulić

Modeling Hierarchical Reasoning Chains by Linking Discourse Units and Key Phrases for Reading Comprehension (2022.coling-1)

Challenge:	Adapters and sparse fine-tuning have been developed to improve transfer learning . a number of approaches have been proposed to improve performance of fine-untuners .
Approach:	They propose a method that fine-tunes the entire set of parameters of a large pretrained model . they use adapters and sparse fine-uning to improve model efficiency .
Outcome:	The proposed method outperforms adapters in cross-lingual transfer benchmarks.

Copied to clipboard

Jialin Chen, Zhuosheng Zhang, Hai Zhao

PFDial: A Structured Dialogue Instruction Fine-tuning Method Based on UML Flowcharts (2025.findings-acl)

Challenge:	Existing methods of logical reasoning focus on entity-aware information but ignore hierarchical relations that may even have mutual effects.
Approach:	They propose a holistic graph network that deals with context at both discourse-level and word-level as the basis for logical reasoning.
Outcome:	The proposed method improves on logical reasoning QA datasets and natural language inference datasets.

Copied to clipboard

Ming Zhang, Yuhui Wang, Yujiong Shen, Tingyi Yang, Changhao Jiang, Yilong Wu, Shihan Dou, Qinhao Chen, Zhiheng Xi, Zhihao Zhang, Yi Dong, Zhen Wang, Zhihui Fei, Mingyang Wan, Tao Liang, Guojun Ma, Qi Zhang, Tao Gui, Xuanjing Huang

Dialogue Systems & Conversational Agents

Challenge:	Large Language Models (LLMs) have shown remarkable progress in dialogue and reasoning, but they struggle to solve strictly constrained dialogue tasks.
Approach:	They construct a dataset that contains 12,705 high-quality Chinese dialogue instructions from 440 flowcharts containing 5,055 process nodes.
Outcome:	The proposed model outperforms GPT-4o models on backward transitions and outperformed GPT-42 models on the same dataset.

A Retrospective Analysis of the Fake News Challenge Stance-Detection Task (C18-1)

Copied to clipboard

Andreas Hanselowski, Avinesh PVS, Benjamin Schiller, Felix Caspelherr, Debanjan Chaudhuri, Christian M. Meyer, Iryna Gurevych

Text Classification

Challenge:	2017 Fake News Challenge Stage 1 (FNC-1) shared task addressed a stance classification task as a crucial first step towards detecting fake news.
Approach:	They propose a new evaluation metric favoring the majority class, which can be easily classified, and propose stacked LSTM models that perform on par with the best systems, but is superior in predicting minority classes.
Outcome:	The proposed evaluation metric favors the majority class, which can be easily classified, and overestimates the true discriminative power of the methods.

Scaling LLMs’ Social Reasoning: Sprinkle Cognitive “Aha Moment” into Fundamental Long-thought Logical Capabilities (2025.findings-acl)

Copied to clipboard

Guiyang Hou, Wenqi Zhang, Zhe Zheng, Yongliang Shen, Weiming Lu

A Dog Is Passing Over The Jet? A Text-Generation Dataset for Korean Commonsense Reasoning and Evaluation (2022.findings-naacl)

Challenge:	Existing studies have examined how large language models’ social reasoning capabilities evolve during model size scaling or reasoning tokens scaling.
Approach:	They propose to optimize evaluation of Large Language Models from both data and model perspectives and to analyze their reasoning trajectories to identify notable cognitive "Aha Moments"
Outcome:	The proposed model outperforms the o1-preview model by 19.0 points in the evaluation of large language models.

Copied to clipboard

Jaehyung Seo, Seounghoon Lee, Chanjun Park, Yoonna Jang, Hyeonseok Moon, Sugyeong Eo, Seonmin Koo, Heuiseok Lim

Exploring Variation of Results from Different Experimental Conditions (2023.findings-acl)

Challenge:	Korean pretrained language models struggle to generate short sentences with a given condition based on compositionality and commonsense reasoning.
Approach:	They propose a Korean text-generation dataset for Korean generative commonsense reasoning and language model evaluation using a semi-automatic dataset construction approach.
Outcome:	The proposed dataset is available at http://aihub.or.kr/opendata/korea-university.

Copied to clipboard

Maja Popović, Mohammad Arvan, Natalie Parde, Anya Belz

Challenge:	Recent research has shown that reproducibility of NLP experiments is not guaranteed by arbitrary factors like random seed and different data splits.
Approach:	They propose to run the training and testing pipeline for three neural text simplification models under varying experimental conditions using the same data and train/dev/test set splits.
Outcome:	The proposed pipeline yields a large number of results for three neural text simplification models under varying experimental conditions.

SocREval: Large Language Models with the Socratic Method for Reference-free Reasoning Evaluation (2024.findings-naacl)

Copied to clipboard

Hangfeng He, Hongming Zhang, Dan Roth

Understanding HTML with Large Language Models (2023.findings-emnlp)

Challenge:	Existing reference-free reasoning evaluation metrics rely on human-annotated reasoning chains as references, but require fine-tuning with human-derived chains before evaluation.
Approach:	They propose to use GPT-4 to automatically evaluate reasoning chain quality by leveraging the Socratic method.
Outcome:	Empirical results show that the proposed approach significantly improves existing reference-free reasoning evaluation metrics.

Copied to clipboard

Izzeddin Gur, Ofir Nachum, Yingjie Miao, Mustafa Safdari, Austin Huang, Aakanksha Chowdhery, Sharan Narang, Noah Fiedel, Aleksandra Faust

A Preference-driven Paradigm for Enhanced Translation with Large Language Models (2024.naacl-long)

Challenge:	Large language models have shown exceptional performance on a variety of natural language tasks, but their capabilities for HTML understanding have not been fully explored.
Approach:	They propose to use HTML understanding models to parse HTML and perform HTML navigation tasks with a large-scale HTML dataset.
Outcome:	The proposed models perform 50% more tasks with 192x less data than the previous best supervised model.

Copied to clipboard

Dawei Zhu, Sony Trenous, Xiaoyu Shen, Dietrich Klakow, Bill Byrne, Eva Hasler

CodeFlowBench: A Multi-turn, Iterative Benchmark for Complex Code Generation (2026.acl-long)

Challenge:	Recent research shows that large language models (LLMs) can achieve remarkable translation performance through supervised fine-tuning (SFT) however, SFT simply instructs the model to imitate reference translations token by token, making it vulnerable to the noise present in the data.
Approach:	They propose a preference-based approach to supervised fine-tuning that trains the model to imitate reference translations token by token, making it vulnerable to noise.
Outcome:	The proposed approach overcomes the plateau associated with imitation-based SFT and is more resilient in the absence of gold translations.

Copied to clipboard

Sizhe Wang, Zhengren Wang, Dongsheng Ma, Yongan Yu, Rui Ling, Zhiyu li, Feiyu Xiong, Wentao Zhang

Programming Languages in NLP

Challenge:	Modern software development demands code that is maintainable, testable, and scalable by organizing the implementation into modular components with iterative reuse of existing codes.
Approach:	They propose a benchmark to evaluate LLMs' ability to perform codeflow by reusing existing functions over multiple turns.
Outcome:	The proposed benchmarks show that LLMs perform significantly worse in multi-turn codeflow scenarios and that their performance inversely correlates with dependency complexity.

Visual Evidence Prompting Mitigates Hallucinations in Large Vision-Language Models (2025.acl-long)

Copied to clipboard

Wei Li, Zhen Huang, Houqiang Li, Le Lu, Yang Lu, Xinmei Tian, Xu Shen, Jieping Ye

Understanding and Addressing the Under-Translation Problem from the Perspective of Decoding Objective (2024.acl-long)

Challenge:	LVLMs have shown impressive progress by integrating visual perception with linguistic understanding to produce contextually grounded outputs.
Approach:	They propose a visual evidence prompting method to mitigate hallucinations in large vision-language models by using small visual models to complement them.
Outcome:	The proposed method reduces hallucinations by reducing false activation and enhancing correct ones.

Copied to clipboard

Chenze Shao, Fandong Meng, Jiali Zeng, Jie Zhou

BELLE: A Bi-Level Multi-Agent Reasoning Framework for Multi-Hop Question Answering (2025.acl-long)

Challenge:	Neural Machine Translation (NMT) has made remarkable progress over the past years, but under-translation and over-translatation remain challenging obstacles faced by NMT systems.
Approach:	They propose to employ the confidence of predicting the end of sentence (EOS) as a detector for under-translation and strengthen the confidence-based penalty to penalize candidates with a high risk of under-translated.
Outcome:	The proposed method can detect and rectify under-translated outputs, with minor impact on other correct translations.

Copied to clipboard

Taolin Zhang, Dongyang Li, Qizhou Chen, Chengyu Wang, Xiaofeng He

Question Answering

Challenge:	Existing studies on multi-hop question answering employ specific methods regardless of question types . complexity of multihop question answerrs often exceeds knowledge boundaries of LLMs .
Approach:	They propose a framework that uses chain-of-thought prompting to prompt LLMs to answer multi-hop questions.
Outcome:	The proposed framework outperforms baseline models in multi-hop QA scenarios.

Revisiting the Gold Standard: Grounding Summarization Evaluation with Robust Human Evaluation (2023.acl-long)

Copied to clipboard

Yixin Liu, Alex Fabbri, Pengfei Liu, Yilun Zhao, Linyong Nan, Ruilin Han, Simeng Han, Shafiq Joty, Chien-Sheng Wu, Caiming Xiong, Dragomir Radev

ForceReader: a BERT-based Interactive Machine Reading Comprehension Model with Attention Separation (2020.coling-main)

Challenge:	Existing studies for summarization evaluation exhibit low inter-annotator agreement or lack scale.
Approach:	They propose a modified summarization salience protocol based on fine-grained semantic units and a robust summarizing evaluation benchmark.
Outcome:	The proposed protocol is based on fine-grained semantic units and allows for high inter-annotator agreement.

Copied to clipboard

Zheng Chen, Kangjian Wu

MMEvalPro: Calibrating Multimodal Benchmarks Towards Trustworthy and Efficient Evaluation (2025.naacl-long)

Challenge:	Various BERT-based reading comprehension models have been proposed, however, these models employ the combined input method without further modification for reading comprehension.
Approach:	They propose a BERT-based interactive machine reading comprehension model that uses BERT's combined input method without further modification for reading comprehension.
Outcome:	The proposed model improves reading comprehension tasks compared to BERT-based models.

Copied to clipboard

Jinsheng Huang, Liang Chen, Taian Guo, Fu Zeng, Yusheng Zhao, Bohan Wu, Ye Yuan, Haozhe Zhao, Zhihui Guo, Yichi Zhang, Jingyang Yuan, Wei Ju, Luchen Liu, Tianyu Liu, Baobao Chang, Ming Zhang

Language Models as Knowledge Bases? (D19-1)

Challenge:	Large Multimodal Models (LMMs) exhibit impressive cross-modal understanding and reasoning abilities, but many benchmarks suffer from systematic biases.
Approach:	They propose a benchmark to avoid Type-I errors by creating one perception question and one knowledge anchor question through a meticulous annotation process.
Outcome:	The proposed benchmark avoids Type-I errors while maintaining reliability of MCQ evaluations.

Copied to clipboard

Fabio Petroni, Tim Rocktäschel, Sebastian Riedel, Patrick Lewis, Anton Bakhtin, Yuxiang Wu, Alexander Miller

ConRAS: Contrastive In-context Learning Framework for Retrieval-Augmented Summarization (2026.findings-eacl)

Challenge:	Recent advances in pretraining language models on large textual corpora led to a surge of improvements for downstream NLP tasks.
Approach:	They present a method for pretraining language models on large textual corpora . they find that they can store relational knowledge and answer queries structured as "fill-in-the-blank" queries.
Outcome:	The proposed language models can recall factual knowledge without fine-tuning without fine tuning . the proposed models can answer queries structured as "fill-in-the-blank" cloze statements .

Copied to clipboard

Juseon Do, Sungwoo Han, Jingun Kwon, Hidetaka Kamigaito, Manabu Okumura

Low-Resource NLP

Challenge:	Despite progress in natural language processing, the potential of contrastive learning remains unexplored.
Approach:	They propose a framework that injects contrastive objectives into in-context learning-based retrieval-augmented summarization.
Outcome:	The proposed framework outperforms state-of-the-art retrieval-augmented methods on three summarization benchmarks showing that it can distinguish between positive and negative samples without parameter updates.

Beyond task success: A closer look at jointly learning to see, ask, and GuessWhat (N19-1)

Copied to clipboard

Ravi Shekhar, Aashish Venkatesh, Tim Baumgärtner, Elia Bruni, Barbara Plank, Raffaella Bernardi, Raquel Fernández

ChainLM: Empowering Large Language Models with Improved Chain-of-Thought Prompting (2024.lrec-main)

Challenge:	Existing systems that address the abilities that need to be put to work during conversations are lacking in terms of visual grounding.
Approach:	They propose a visually-grounded dialogue state encoder which integrates visual grounding with dialogue system components.
Outcome:	The proposed system improves the GuessWhat?! game by combining guessing and asking questions with multi-task learning.

Copied to clipboard

Xiaoxue Cheng, Junyi Li, Wayne Xin Zhao, Ji-Rong Wen

Lost in Transcription: Identifying and Quantifying the Accuracy Biases of Automatic Speech Recognition Systems Against Disfluent Speech (2024.naacl-long)

Challenge:	Existing CoT synthesis approaches focus on simpler reasoning tasks and result in inconsistent CoT prompts.
Approach:	They propose a framework for automatic generation of superior CoT prompts based on three major evolution strategies . they propose 'step-level debating' method where multiple debaters discuss each reasoning step to arrive at the correct answer.
Outcome:	The proposed framework can generate superior CoT prompts from a CoT dataset.

Copied to clipboard

Dena Mujtaba, Nihar Mahapatra, Megan Arney, J Yaruss, Hope Gerlach-Houck, Caryn Herring, Jia Bin

No Context Needed: Contextual Quandary In Idiomatic Reasoning With Pre-Trained Language Models (2024.naacl-long)

Challenge:	Automatic speech recognition systems fail to accurately interpret speech patterns deviating from typical fluency, leading to critical usability issues and misinterpretations.
Approach:	They evaluate six leading automatic speech recognition systems based on a real-world dataset and a synthetic dataset derived from the widely-used LibriSpeech benchmark.
Outcome:	The six leading speech recognition systems were evaluated on a real-world dataset and a synthetic dataset derived from the widely-used LibriSpeech benchmark.

Copied to clipboard

Kellen Cheng, Suma Bhat

Learning Better Internal Structure of Words for Sequence Labeling (D18-1)

Challenge:	idiomatic expressions (IEs) are a non-compositional aspect of a text that makes it difficult for a model to comprehend . general purpose PTLMs are negatively affected by the context, as performance increases with its removal.
Approach:	They propose to use idiomatic expressions to infer additional meaning from IEs . they argue that only IE-aware models are suitable for idiom- matic reasoning tasks .
Outcome:	The proposed models can reason in the presence of idiomatic expressions, the authors show . they show that general purpose PTLMs are negatively affected by the context .

Copied to clipboard

Yingwei Xin, Ethan Hart, Vibhuti Mahajan, Jean-David Ruvini

Representation Learning

Challenge:	a gap exists between methods for learning representations of sentences and words . authors propose a convolutional neural architecture with no down-sampling for learning words based on character embeddings .
Approach:	They propose a funnel-shaped wide convolutional neural architecture with no down-sampling for learning words' internal structure.
Outcome:	The proposed model outperforms other character embedding models on six sequence labeling datasets.

An Evaluation Method for Diachronic Word Sense Induction (2020.findings-emnlp)

Copied to clipboard

Ashjan Alsulaimani, Erwan Moreau, Carl Vogel

Challenge:	Existing methods to detect semantic shifts across time are based on time-stamped annotated biomedical data . dynamic behaviour of words contributes to semantic ambiguity, which is a challenge in many NLP tasks.
Approach:	They propose an evaluation method based on large-scale time-stamped biomedical data . they propose a model which represents the temporal dimension of the task .
Outcome:	The proposed method is applied to two recent DWSI systems . it provides an in-depth analysis of the models .

Self-Tuning: Instructing LLMs to Effectively Acquire New Knowledge through Self-Teaching (2025.findings-acl)

Copied to clipboard

Xiaoying Zhang, Baolin Peng, Ye Tian, Jingyan Zhou, Yipeng Zhang, Haitao Mi, Helen M. Meng

CARFT: Boosting LLM Reasoning via Contrastive Learning with Annotated Chain-of-Thought-based Reinforced Fine-Tuning (2025.emnlp-main)

Challenge:	Existing approaches to keeping large language models current involve continued pre-training on new documents.
Approach:	They propose a learning framework that augments documents with knowledge-intensive tasks created in a self-supervised manner, focusing on memorization, comprehension, and self-reflection.
Outcome:	The proposed learning framework improves an LLM’s ability to acquire new knowledge from unseen raw documents through self-teaching.

Copied to clipboard

Wenqiao Zhu, Ji Liu, Rongjunchen Zhang, Haipang Wu, Yulun Zhang

McBE: A Multi-task Chinese Bias Evaluation Benchmark for Large Language Models (2025.findings-acl)

Challenge:	Existing methods to improve the reasoning performance of Large Language Models (LLMs) ignore annotated Chain-of-Thought (CoT) and incorporate unstable reasoning path sampling.
Approach:	They propose a Contrastive learning with annotated CoT-based Reinforced Fine-Tuning approach to enhance the reasoning performance of Large Language Models.
Outcome:	The proposed approach exploits annotated CoT and stabilizes the fine-tuning procedure by incorporating an additional unsupervised learning signal.

Copied to clipboard

Tian Lan, Xiangdong Su, Xu Liu, Ruirui Wang, Ke Chang, Jiang Li, Guanglai Gao

Decomposition Dilemmas: Does Claim Decomposition Boost or Burden Fact-Checking Performance? (2025.naacl-long)

Challenge:	Existing datasets on bias evaluation for large language models focus on English and North American culture and are limited to one task.
Approach:	They propose to evaluate Chinese language models' biases from multiple perspectives using a multi-task Chinese Bias Evaluation Benchmark.
Outcome:	The proposed model covers 12, 82 subcategories and 5 evaluation tasks covering a wide range of categories and content diversity.

Copied to clipboard

Qisheng Hu, Quanyu Long, Wenya Wang

Analyzing Key Factors Influencing Emotion Prediction Performance of VLLMs in Conversational Contexts (2024.emnlp-main)

Challenge:	Fact-checking pipelines adopt the Decompose-Then-Verify paradigm, where texts are broken down into smaller claims for individual verification and subsequently combined for a veracity decision.
Approach:	They propose to categorize decomposition errors and to reveal a trade-off between accuracy gains and noise introduced by decomposing.
Outcome:	The proposed analysis provides new insights into understanding current system’s instability and offers guidance for future studies toward improving claim decomposition in fact-checking pipelines.

Copied to clipboard

Jaewook Lee, Yeajin Jang, Hongjin Kim, Woojin Lee, Harksoo Kim

When Allies Turn Foes: Exploring Group Characteristics of LLM-Based Multi-Agent Collaborative Systems Under Adversarial Attacks (2025.findings-emnlp)

Challenge:	Recent studies show that large language models and vision large language model (VLLMs) possess EI and the ability to understand emotional stimuli in the form of text and images.
Approach:	They analyze the key elements affecting the emotion prediction performance of VLLMs in conversational contexts.
Outcome:	The proposed model performance was compared with other models in a conversational context.

Copied to clipboard

Jiahao Zhang, Baoshuo Kan, Tao Gong, Fu Lee Wang, Tianyong Hao

FastFiD: Improve Inference Efficiency of Open Domain Question Answering via Sentence Selection (2024.acl-long)

Challenge:	a new study examines the group characteristics of adversarial agents in multi-agent collaborative systems . collaborative agents are tasked with generating counterfactual answers to a given collaborative problem .
Approach:	They evaluate collaborative systems under adversarial attacks and propose methods to mitigate them . they also introduce a new metric to quantify the robustness of collaborative systems against such attacks .
Outcome:	The proposed method has been proven effective against adversarial attacks.

Copied to clipboard

Yufei Huang, Xu Han, Maosong Sun

LibEvolutionEval: A Benchmark and Study for Version-Specific Code Generation (2025.naacl-long)

Challenge:	Open Domain Question Answering (ODQA) is a longstanding task in Natural Language Processing that involves generating an answer solely based on a given question.
Approach:	They propose a novel approach that executes sentence selection on the encoded passages to enhance the inference speed while reducing the context length required for generating answers.
Outcome:	The proposed approach can increase inference speed by 2.3X-5.7X while maintaining the model’s performance.

Copied to clipboard

Sachit Kuhar, Wasi Uddin Ahmad, Zijian Wang, Nihal Jain, Haifeng Qian, Baishakhi Ray, Murali Krishna Ramanathan, Xiaofei Ma, Anoop Deoras

Programming Languages in NLP

Challenge:	Recent code completion models focus on local file contexts, but do not fully capture the complexities of real-world software development.
Approach:	They propose a version-specific code-completion task across eight libraries as they evolve over the years and an in-depth analysis of two widely used public libraries: PyTorch and Matplotlib.
Outcome:	The proposed model improves performance with public libraries, compared with existing models.

Self-Correction Makes LLMs Better Parsers (2025.findings-emnlp)

Copied to clipboard

Ziyan Zhang, Yang Hou, Chen Gong, Zhenghua Li

Syntactic Text Processing

Challenge:	Large language models (LLMs) have achieved remarkable success across various natural language processing tasks, but they still face challenges in performing fundamental NLP tasks, such as syntactic parsing.
Approach:	They propose a method that leverages grammar rules from existing treebanks to guide LLMs in correcting previous errors.
Outcome:	The proposed method significantly improves performance on in-domain and cross-domain datasets.

Large-Scale and Multi-Perspective Opinion Summarization with Diverse Review Subsets (2023.findings-emnlp)

Copied to clipboard

Han Jiang, Rui Wang, Zhihua Wei, Yu Li, Xinpeng Wang

Text Generation

Challenge:	Existing methods for opinion summarization are deficient in epitomizing extensive reviews and offering opinion summaries from various angles.
Approach:	They propose a supervised opinion summarization framework that takes sentiment orientation into account and trains the summarizer to learn from sub-optimal and optimal review subsets.
Outcome:	The proposed framework generates pros, cons, and verdict summaries from hundreds of input reviews.

On the Effect of Pretraining Corpora on In-context Learning by a Large-scale Language Model (2022.naacl-main)

Copied to clipboard

Seongjin Shin, Sang-Woo Lee, Hwijeen Ahn, Sungdong Kim, HyoungSeok Kim, Boseop Kim, Kyunghyun Cho, Gichang Lee, Woomyoung Park, Jung-Woo Ha, Nako Sung

ESG-Kor: A Korean Dataset for ESG-related Information Extraction and Practical Use Cases (2024.findings-emnlp)

Challenge:	Recent studies on large-scale in-context language models have reported successful in-const zero- and few-shot learning ability.
Approach:	They investigate the effects of the pretraining corpus on in-context learning in a Korean-centric model.
Outcome:	The study shows that pretraining corpus size does not determine in-context learning ability . the findings suggest that in-constext learning is not always competitive .

Copied to clipboard

Jaeyoung Lee, Geonyeong Son, Misuk Kim

Information Extraction & Text Mining

Challenge:	Pre-trained language models are exhibiting astonishing performances in various natural language processing tasks, including classification, question answering, machine translation, summarization, and conversation generation.
Approach:	They built a Korean dataset to automatically extract Environmental, Social, and Governance (ESG) information from Korean companies’ sustainability reports and manually labeled it according to objective rules provided by ESG evaluation agencies.
Outcome:	The proposed dataset extracts environmental, social, and governance information from Korean companies’ sustainability reports and labels it according to objective rules provided by ESG evaluation agencies.

What Prompts Don’t Say: Understanding and Managing Underspecification in LLM Prompts (2026.findings-acl)

Copied to clipboard

Chenyang Yang, Yike Shi, Qianou Ma, Michael Xieyang Liu, Christian Kaestner, Tongshuang Wu

On the Robustness of Cognate Generation Models (2022.lrec-1)

Challenge:	Under-specified prompts are 2x as likely to regress across model or prompt changes, authors show . eliot safina: a lack of explicit prompts can cause frustrations and failures .
Approach:	They propose requirements-aware prompt optimization mechanisms that improve performance by 4.8% over baselines.
Outcome:	The proposed mechanisms improve prompt performance by 4.8% over baselines.

Copied to clipboard

Winston Wu, David Yarowsky

MultiFC: A Real-World Multi-Domain Dataset for Evidence-Based Fact Checking of Claims (D19-1)

Challenge:	We examine different types of noise generated by human errors and how these noisy inputs affect the performance of cognate generation models.
Approach:	They evaluate two popular neural cognate generation models’ robustness to human-plausible noise.
Outcome:	The proposed models are robust to deletion, duplication, swapping, keyboard errors, and a new type of error, phonological errors.

Copied to clipboard

Isabelle Augenstein, Christina Lioma, Dongsheng Wang, Lucas Chaves Lima, Casper Hansen, Christian Hansen, Jakob Grue Simonsen

Polishing Every Facet of the GEM: Testing Linguistic Competence of LLMs and Humans in Korean (2025.acl-long)

Challenge:	Existing efforts to verify factual claims are limited by small datasets or artificially constructed datasets.
Approach:	They propose to use the largest publicly available dataset of naturally occurring factual claims for automatic claim verification.
Outcome:	The proposed model outperforms baseline models and evidence pages significantly.

Copied to clipboard

SungHo Kim, Nayeon Kim, Taehee Jeon, SangKeun Lee

Challenge:	Existing studies have focused on linguistic competence of language models with grammatical knowledge.
Approach:	They propose to use grammar as a measurable proxy to assess linguistic competence of large language models (LLMs) .
Outcome:	The proposed model aims to assess the linguistic competence of large language models (LLMs) and humans in Korean.

ReCaLL: Membership Inference via Relative Conditional Log-Likelihoods (2024.emnlp-main)

Copied to clipboard

Roy Xie, Junlin Wang, Ruomin Huang, Minxing Zhang, Rong Ge, Jian Pei, Neil Gong, Bhuwan Dhingra

Eliciting Motivational Interviewing Skill Codes in Psychotherapy with LLMs: A Bilingual Dataset and Analytical Study (2024.lrec-main)

Challenge:	ReCaLL (Relative Conditional Log-Likelihood) is a membership inference attack that can detect LLMs’ pretraining data by leveraging their conditional language modeling capabilities.
Approach:	They propose a membership inference attack to detect LLMs’ pretraining data by leveraging their conditional language modeling capabilities.
Outcome:	The proposed model achieves state-of-the-art performance on the WikiMIA dataset, even with random and synthetic prefixes, and can be further improved using an ensemble approach.

Copied to clipboard

Xin Sun, Jiahuan Pei, Jan de Wit, Mohammad Aliannejadi, Emiel Krahmer, Jos T.P. Dobber, Jos A. Bosch

IRAC: A Domain-Specific Annotated Corpus of Implicit Reasoning in Arguments (2022.lrec-1)

Challenge:	Motivational interviewing (MI) is an essential, directive, client-centered counseling technique.
Approach:	They propose a bilingual dataset of MI conversations in English and Dutch . they propose an approach to elicit MISC expertise from Large language models .
Outcome:	The proposed approach yields results aligned with expert annotations and maintains consistent performance across languages.

Copied to clipboard

Keshav Singh, Naoya Inoue, Farjana Sultana Mim, Shoichi Naito, Kentaro Inui

Benchmarking Large Language Models Under Data Contamination: A Survey from Static to Dynamic Evaluation (2025.emnlp-main)

Challenge:	Using crowdsourcing, we show that models trained with domain-specific implicit reasonings outperform domain-general models in both automatic and human evaluations.
Approach:	They propose to create a domain-specific corpus of implicit reasonings annotated for a wide range of arguments and use it to generate models.
Outcome:	The proposed corpus outperforms domain-general models in automatic and human evaluations.

Copied to clipboard

Simin Chen, Yiming Chen, Zexin Li, Yifan Jiang, Zhongwei Wan, Yixin He, Dezhi Ran, Tianle Gu, Haizhou Li, Tao Xie, Baishakhi Ray

Native-like Expression Identification by Contrasting Native and Proficient Second Language Speakers (2020.coling-main)

Challenge:	In the era of evaluating large language models, data contamination is an increasingly prominent concern . static benchmarking has been used for evaluation, but there are limitations of dynamic benchmarks .
Approach:	They propose a series of optimal design principles for dynamic benchmarking and analyze the limitations of existing static benchmarks.
Outcome:	The proposed benchmarks highlight a critical gap in the evaluation of LLMs.

Copied to clipboard

Oleksandr Harust, Yugo Murawaki, Sadao Kurohashi

Text Classification

Challenge:	a novel task of native-like expression identification is proposed by contrasting texts written by native speakers and those by proficient second language speakers.
Approach:	They propose a task of native-like expression identification by contrasting texts written by native speakers and those by proficient second language speakers.
Outcome:	The proposed method uncovers linguistically interesting usages distinctive of native speech.

T-Eval: Evaluating the Tool Utilization Capability of Large Language Models Step by Step (2024.acl-long)

Copied to clipboard

Zehui Chen, Weihua Du, Wenwei Zhang, Kuikun Liu, Jiangning Liu, Miao Zheng, Jingming Zhuo, Songyang Zhang, Dahua Lin, Kai Chen, Feng Zhao

LINKED: Eliciting, Filtering and Integrating Knowledge in Large Language Model for Commonsense Reasoning (2024.findings-emnlp)

Challenge:	Existing studies evaluate the tool utilization ability of large language models based on the final output or only consider the single-step tool calling.
Approach:	They propose a new approach to evaluate the tool utilization capability of large language models (LLMs) they decompose the tool usage into multiple sub-processes, including instruction following, planning, reasoning, retrieval, understanding, and review.
Outcome:	The proposed model exhibits consistency with the outcome-oriented evaluation and provides a more fine-grained analysis of the capabilities of LLMs.

Copied to clipboard

Jiachun Li, Pengfei Cao, Chenhao Wang, Zhuoran Jin, Yubo Chen, Kang Liu, Xiaojian Jiang, Jiexin Xu, Jun Zhao

Multilingual Dictionary Based Construction of Core Vocabulary (2020.lrec-1)

Challenge:	Large language models (LLMs) often exhibit poor performance on knowledge-intensive tasks, such as commonsense reasoning.
Approach:	They propose a method to elicit, filter and integrate knowledge in large language models (LINKED) they propose 'reward model' to filter out noisy knowledge and 'take marginal consistent reasoning module'
Outcome:	The proposed method outperforms SOTA baselines on two commonsense reasoning tasks.

Copied to clipboard

Winston Wu, Garrett Nicolai, David Yarowsky

Are we Estimating or Guesstimating Translation Quality? (2020.acl-main)

Challenge:	Existing methods for core vocabulary lists for multiple applications are lacking coverage in sparse dictionaries . we propose a new method for definition and construction of core vocabulary sets based on coverage in dictionary dictionaria .
Approach:	They propose a functional definition and construction method for core vocabulary sets based on relative coverage of a target concept in bilingual dictionaries.
Outcome:	The proposed method achieves high overlap with existing vocabulary lists . it uses a cognate prediction method to recover missing coverage of the vocabulary .

Copied to clipboard

Shuo Sun, Francisco Guzmán, Lucia Specia

Towards Better Chain-of-Thought: A Reflection on Effectiveness and Faithfulness (2025.findings-acl)

Challenge:	A carefully engineered ensemble of pre-trained multilingual language models won the QE shared task at WMT19.
Approach:	They propose to use pre-trained multilingual language models to train quality estimation for machine translation.
Outcome:	A carefully engineered ensemble of pre-trained language models wins the QE shared task at WMT19.

Copied to clipboard

Jiachun Li, Pengfei Cao, Yubo Chen, Jiexin Xu, Huaijun Li, Xiaojian Jiang, Kang Liu, Jun Zhao

Challenge:	Chain-of-thought (CoT) prompting demonstrates varying performance under different reasoning tasks.
Approach:	They propose to recall extra information from the question to enhance CoT generation and evaluate CoTs based on their information gain.
Outcome:	The proposed method improves both the faithfulness and effectiveness of CoT and evaluates it based on their information gain.

CrossFit: A Few-shot Learning Challenge for Cross-task Generalization in NLP (2021.emnlp-main)

Copied to clipboard

Qinyuan Ye, Bill Yuchen Lin, Xiang Ren

Biomedical Concept Relatedness – A large EHR-based benchmark (2020.coling-main)

Challenge:	We study whether and how cross-task generalization ability can be acquired . we use CrossFit to standardize seen/unseen task partitions and evaluation protocols .
Approach:	They propose a problem setup for studying cross-task generalization ability which standardizes seen/unseen task partitions and data access during different learning stages.
Outcome:	The proposed model can be used to build few-shot learners across diverse tasks.

Copied to clipboard

Claudia Schulz, Josh Levy-Kramer, Camille Van Assel, Miklos Kepes, Nils Hammerla

Mathematical Proof as a Litmus Test: Revealing Failure Modes of Advanced Large Reasoning Models (2026.acl-long)

Challenge:	Existing biomedical concept relatedness datasets are notoriously small and consist of hand-picked concept pairs.
Approach:	They propose to use a concept relatedness benchmark to test the suitability of AI in healthcare . they find that it is six times larger than existing concepts relatedness datasets .
Outcome:	The proposed benchmark is six times larger than existing biomedical concept relatedness datasets and is relevant for the application of interest.

Copied to clipboard

Dadi Guo, Jiayu Liu, Zhiyuan Fan, Zhitao He, Haoran Li, Yuxin Li, Yumeng Wang, Yi R. Fung

Large Language Models are Miscalibrated In-Context Learners (2025.findings-acl)

Challenge:	Large reasoning models have demonstrated remarkable mathematical problem-solving abilities, but their true reasoning shortcomings are often hidden.
Approach:	They propose to leverage the rigor and methodological complexity of mathematical proofs as a diagnostic tool to expose hidden failures.
Outcome:	The proposed model evaluation exploits the rigor and complexity of proof problems to uncover 10 fine-grained errors.

Copied to clipboard

Chengzu Li, Han Zhou, Goran Glavaš, Anna Korhonen, Ivan Vulić

How Personality Traits Influence Negotiation Outcomes? A Simulation based on Large Language Models (2024.findings-emnlp)

Challenge:	In-context Learning and Supervised Fine-Tuning have emerged as pre-dominant methodologies for machine learning and NLP.
Approach:	They propose to use self-ensembling to improve both performance and calibration of language models.
Outcome:	The proposed learning paradigms can achieve better calibration and better performance than the previous learning paradigm.

Copied to clipboard

Yin Jou Huang, Rafik Hadfi

Language Models Predict Empathy Gaps Between Social In-groups and Out-groups (2025.naacl-long)

Challenge:	Psychological evidence reveals the influence of personality traits on decision-making.
Approach:	They propose a simulation framework centered on large language model agents with synthesized personality traits and propose empirical insights into the strategic impacts of Big Five personality traits on outcomes of bilateral negotiations.
Outcome:	The proposed model can reproduce behavioral patterns observed in human negotiations.

Copied to clipboard

Yu Hou, Hal Daumé Iii, Rachel Rudinger

Few-shot Learning with Multilingual Generative Language Models (2022.emnlp-main)

Challenge:	Studies of human psychology have shown that people are more motivated to extend empathy to in-group members than out-group member.
Approach:	They propose to use language models to study intergroup empathy gap . they use a short description of an experience to predict emotion intensity .
Outcome:	The proposed model exhibited strongest intergroup bias among those tested.

Copied to clipboard

Xi Victoria Lin, Todor Mihaylov, Mikel Artetxe, Tianlu Wang, Shuohui Chen, Daniel Simig, Myle Ott, Naman Goyal, Shruti Bhosale, Jingfei Du, Ramakanth Pasunuru, Sam Shleifer, Punit Singh Koura, Vishrav Chaudhary, Brian O’Horo, Jeff Wang, Luke Zettlemoyer, Zornitsa Kozareva, Mona Diab, Veselin Stoyanov, Xian Li

Exploiting Unlabeled Data for Target-Oriented Opinion Words Extraction (2022.coling-1)

Challenge:	Large-scale generative language models such as GPT-3 are competitive few-shot learners.
Approach:	They train multilingual generative language models on a corpus covering a diverse set of languages and study their few- and zero-shot learning capabilities.
Outcome:	The proposed model outperforms GPT-3 on 171 out of 182 directions with 32 training examples and surpasses the official supervised baseline in 45 directions.

Copied to clipboard

Yidong Wang, Hao Wu, Ao Liu, Wenxin Hou, Zhen Wu, Jindong Wang, Takahiro Shinozaki, Manabu Okumura, Yue Zhang

Low-Resource NLP

Challenge:	Existing methods to extract opinion words from sentences are limited due to the expensive annotation process.
Approach:	They propose to exploit massive unlabeled data to reduce distribution shift risk . they propose to use two filters specifically for TOWE to filter noisy data . results indicate superiority of MGCR over current state-of-the-art methods .
Outcome:	The proposed method reduces the risk of distribution shifts by increasing the exposure of the model to varying distribution shift.

Understanding ME? Multimodal Evaluation for Fine-grained Visual Commonsense (2022.emnlp-main)

Copied to clipboard

Zhecan Wang, Haoxuan You, Yicheng He, Wenhao Li, Kai-Wei Chang, Shih-Fu Chang

Find-the-Common: A Benchmark for Explaining Visual Patterns from Images (2024.lrec-main)

Challenge:	Existing models that understand image and text but also cross-reference in-between are lacking in evaluation data resources.
Approach:	They propose a multimodal evaluation pipeline to automatically generate question-answer pairs to test models’ understanding of the visual scene, text, and related knowledge.
Outcome:	The proposed model can answer the highly semantic VCR question correctly but fails to answer related visual question (Q2), textual question (q3), and background knowledge question ( Q4) as shallow mappings with language priors and unbalanced utilization of information between modalities.

Copied to clipboard

Yuting Shi, Naoya Inoue, Houjing Wei, Yufeng Zhao, Tao Jin

Cross-Lingual Consistency of Factual Knowledge in Multilingual Language Models (2023.emnlp-main)

Challenge:	Recent advances in Instruction-fine-tuned Vision and Language Models (IVLMs) have prompted some studies to analyze the reasoning capabilities of IVLMs.
Approach:	They introduce a vision and language task for Inductive Visual Reasoning that uses common attributes across visual scenes to find common answers.
Outcome:	The proposed model can archive with 48% accuracy on the FTC, compared with state-of-the-art models.

Copied to clipboard

Jirui Qi, Raquel Fernández, Arianna Bisazza

Saving Dense Retriever from Shortcut Dependency in Conversational Search (2022.emnlp-main)

Challenge:	Multilingual large-scale pretrained language models store factual knowledge, but large variations are observed across languages.
Approach:	They propose a ranking-based consistency metric to evaluate cross-lingual consistency of factual knowledge in multilingual PLMs.
Outcome:	The proposed metric evaluates cross-lingual consistency of factual knowledge across languages independently from accuracy.

Copied to clipboard

Sungdong Kim, Gangwoo Kim

Cross-Domain Label-Adaptive Stance Detection (2021.emnlp-main)

Challenge:	a retrieval shortcut in conversational search (CS) relies on partial history to retrieve relevant passages . naively trained dense retrievers heavily exploit the shortcut and perform poorly when asked to answer history-independent questions.
Approach:	They propose to exploit a retrieval shortcut in conversational search (CS) that allows models to only use partial history to retrieve relevant passages while disregarding the latest question.
Outcome:	The proposed model outperforms the previous state-of-the-art model by 11.0 on QReCC.

Copied to clipboard

Momchil Hardalov, Arnav Arora, Preslav Nakov, Isabelle Augenstein

Sentiment Analysis

Challenge:	Stance detection is a task that focuses on the classification of a writer’s viewpoint towards a target.
Approach:	They propose an end-to-end unsupervised framework for out-of-domain prediction of unseen, user-defined labels.
Outcome:	The proposed framework shows that it can be used to predict unseen labels over strong baselines.

How Sememic Components Can Benefit Link Prediction for Lexico-Semantic Knowledge Graphs? (2025.emnlp-main)

Copied to clipboard

Hansi Wang, Yue Wang, Qiliang Liang, Yang Liu

Modeling Subjectivity in Cognitive Appraisal with Language Models (2025.findings-emnlp)

Challenge:	Existing methods to predict missing triples in Knowledge Graphs are limited by semantic information.
Approach:	They propose a method to leverage sememe knowledge to enhance LP . LP is a technique that integrates structural and textual information into a Knowledge Graph .
Outcome:	The proposed method improves LP performance in English and Chinese . it improves on WN18RR, HN7 and CWN5, respectively .

Copied to clipboard

Yuxiang Zhou, Hainiu Xu, Desmond Ong, Maria Liakata, Petr Slovak, Yulan He

Annotation of Valence Unfolding in Spoken Personal Narratives (2022.lrec-1)

Challenge:	a new study explores how language models can quantify subjectivity in cognitive appraisal . existing post-hoc calibration methods fail to achieve satisfactory performance .
Approach:	They investigate how language models can quantify subjectivity in cognitive appraisal . existing post-hoc calibration methods often fail to achieve satisfactory performance .
Outcome:	The proposed model can quantify subjectivity in cognitive appraisal using fine-tuned models and prompt-based large language models.

Copied to clipboard

Aniruddha Tammewar, Franziska Braun, Gabriel Roccabruna, Sebastian Bayerl, Korbinian Riedhammer, Giuseppe Riccardi

Speech & Audio in NLP

Challenge:	Personal Narrative (PN) is the recollection of individuals’ life experiences, events, and thoughts along with the associated emotions in the form of a story.
Approach:	They annotate a corpus of spoken personal narratives with the emotion valence using discrete values and use a 5-point bipolar scale to measure their agreement.
Outcome:	The annotators annotate a corpus of spoken personal narratives with the emotion valence using discrete values on a 5-point bipolar scale ranging from -2 to +2 (0 for neutral).

A Systematic Analysis of Large Language Models as Soft Reasoners: The Case of Syllogistic Inferences (2024.emnlp-main)

Copied to clipboard

Leonardo Bertolazzi, Albert Gatt, Raffaella Bernardi

Pcc-tuning: Breaking the Contrastive Learning Ceiling in Semantic Textual Similarity (2024.emnlp-main)

Challenge:	syllogistic reasoning is a deductive reasoning skill that is crucial in everyday problem-solving and decision-making experiences.
Approach:	They propose to study the reasoning abilities of Large Language Models (LLMs) they propose to use supervised fine-tuning and chain-of-thought reasoning to investigate their results.
Outcome:	The proposed models exhibit reasoning biases, avoid answering that no conclusion follows, align with human difficulties, and struggle with multi-step reasoning.

Copied to clipboard

Bowen Zhang, Chunping Li

Curse of Knowledge: Your Guidance and Provided Knowledge are biasing LLM Judges in Complex Evaluation (2025.findings-emnlp)

Challenge:	Semantic Textual Similarity (STS) is a key indicator of the encoding capabilities of embedding models.
Approach:	They propose to use Pearson’s correlation coefficient as a loss function to refine model performance beyond contrastive learning to achieve a Spearman’s ceiling.
Outcome:	The proposed method surpasses state-of-the-art strategies with minimal amount of fine-grained annotated samples.

Copied to clipboard

Weiyuan Li, Xintao Wang, Siyu Yuan, Rui Xu, Jiangjie Chen, Qingqing Dong, Yanghua Xiao, Deqing Yang

Challenge:	a recent study has focused on simple settings, but their reliability in complex tasks remains understudied.
Approach:	They propose to use large language models as judges to evaluate reliability in complex tasks . they use a challenge benchmark to expose and quantify Auxiliary Information Induced Biases .
Outcome:	The proposed benchmark exposes and quantifies Auxiliary Information Induced Biases across 12 basic and 3 advanced scenarios.

OpenCQA: Open-ended Question Answering with Charts (2022.emnlp-main)

Copied to clipboard

Shankar Kantharaj, Xuan Long Do, Rixie Tiffany Leong, Jia Qing Tan, Enamul Hoque, Shafiq Joty

Question Answering

Challenge:	OpenCQA is a task to answer open-ended questions about charts with descriptive texts.
Approach:	They propose a task to answer open-ended questions about charts with descriptive texts.
Outcome:	The proposed task is to answer an open-ended question about a chart with descriptive texts.

Can LLMs Replace Clinical Doctors? Exploring Bias in Disease Diagnosis by Large Language Models (2024.findings-emnlp)

Copied to clipboard

Yutian Zhao, Huimin Wang, Yuqi Liu, Wu Suhuang, Xian Wu, Yefeng Zheng

Movie101v2: Improved Movie Narration Benchmark (2025.acl-long)

Challenge:	a new study examines the bias of disease prediction in large language models . the model biases are prevalent across gender, age range and disease judgment behaviors .
Approach:	They propose a prompt-based approach to alleviate the bias in disease prediction with LLMs.
Outcome:	The proposed model alleviates the observed bias in disease prediction with LLMs.

Copied to clipboard

Zihao Yue, Yepeng Zhang, Ziheng Wang, Qin Jin

Visual Data in NLP

Challenge:	Automatic movie narration aims to generate video-aligned plot descriptions to assist visually impaired audiences.
Approach:	They propose to break down the ultimate goal of automatic movie narration into three stages . they propose a large-scale, bilingual dataset with enhanced data quality .
Outcome:	The proposed dataset breaks down the goal of automatic movie narration into three stages . achieving applicable movie narration is a fascinating goal that requires significant research .

Syllogistic Reasoning for Legal Judgment Analysis (2023.emnlp-main)

Copied to clipboard

Wentao Deng, Jiahuan Pei, Keyi Kong, Zhe Chen, Furu Wei, Yujun Li, Zhaochun Ren, Zhumin Chen, Pengjie Ren

The Anatomy of Evidence: An Investigation Into Explainable ICD Coding (2025.findings-acl)

Challenge:	Legal judgment assistants are developing fast due to impressive progress of large language models.
Approach:	They construct and manually correct a syllogistic reasoning dataset for legal judgment analysis using large language models as benchmarks.
Outcome:	The proposed dataset contains 11,239 criminal cases covering 4 criminal elements, 80 charges and 124 articles.

Copied to clipboard

Katharina Beckh, Elisa Studeny, Sujan Sai Gannamaneni, Dario Antweiler, Stefan Rueping

LFED: A Literary Fiction Evaluation Dataset for Large Language Models (2024.lrec-main)

Challenge:	Existing systems that use explainability methods have limited their effectiveness due to a lack of annotated data.
Approach:	They propose match measures and highlight success and failure cases to improve automatic medical coding.
Outcome:	The proposed models show that ground truth evidence aligns with code descriptions to a certain degree.

Copied to clipboard

Linhao Yu, Qun Liu, Deyi Xiong

LHMKE: A Large-scale Holistic Multi-subject Knowledge Evaluation Benchmark for Chinese Large Language Models (2024.lrec-main)

Challenge:	LFED is a literary fiction evaluation dataset for large language models that evaluate the capability of LLMs on the long fiction comprehension and reasoning.
Approach:	They propose a Literary Fiction Evaluation Dataset to evaluate LLMs' comprehension and reasoning on long fictions.
Outcome:	The proposed dataset evaluates the capability of large language models on the long fiction comprehension and reasoning.

Copied to clipboard

Chuang Liu, Renren Jin, Yuqi Ren, Deyi Xiong

LexGenius: An Expert-Level Benchmark for Large Language Models in Legal General Intelligence (2026.findings-acl)

Challenge:	Existing benchmarks for comprehensively evaluating Chinese Large Language Models are insufficient.
Approach:	They propose a Large-scale, Holistic, and Multi-subject Knowledge Evaluation benchmark to evaluate Chinese Large Language Models.
Outcome:	The proposed benchmark measures the knowledge acquisition capabilities of Chinese Large Language Models across 75 subjects from primary school to professional certification exams.

Copied to clipboard

Wenjin Liu, Haoran Luo, Xin Feng, Xiang Ji, Lijuan Zhou, Rui Mao, Jiapu Wang, Shirui Pan, Erik Cambria

Explicit Memory Learning with Expectation Maximization (2024.emnlp-main)

Challenge:	Existing benchmarks for legal general intelligence (GI) are result-oriented and do not evaluate the legal intelligence of large language models (LLMs).
Approach:	They propose a Chinese legal benchmark for evaluating legal GI in large language models . they use recent legal cases and exam questions to create multiple-choice questions .
Outcome:	The proposed benchmarks lack a systematic evaluation of the legal intelligence of large language models (LLMs) the results show that even the best LLMs lagging behind human legal professionals.

Copied to clipboard

Zhangyue Yin, Qiushi Sun, Qipeng Guo, Zhiyuan Zeng, Qinyuan Cheng, Xipeng Qiu, Xuanjing Huang

Challenge:	Large Language Models lack reliable learning mechanisms for updating information across interactions.
Approach:	They propose a framework that enhances explicit memory updates via the Expectation-Maximization algorithm.
Outcome:	The proposed framework outperforms existing methods without memory or with static external memory on streaming inference tasks.

VIVA+: Human-Centered Situational Decision-Making (2025.findings-emnlp)

Copied to clipboard

Zhe Hu, Yixiao Ren, Guanzhong Liu, Jing Li, Yu Yin

Sowing the Wind, Reaping the Whirlwind: The Impact of Editing Language Models (2024.findings-acl)

Challenge:	Multimodal Large Language Models (MLLMs) show promising results in complex, human-centered environments, yet evaluating their capacity for nuanced, humanlike reasoning and decision-making remains challenging.
Approach:	They introduce VIVA+, a cognitively grounded benchmark for evaluating the reasoning and decision-making of MLLMs in human-centered situations.
Outcome:	The VIVA+ model is based on 1,317 real-world situations paired with 6,373 multiple-choice questions . it consists of three core abilities for decision-making: (1) Foundational Situation Comprehension, (2) Context-Driven Action Justification, and (3) Reflective Reasoning.

Copied to clipboard

Rima Hazra, Sayan Layek, Somnath Banerjee, Soujanya Poria

Improving Input-label Mapping with Demonstration Replay for In-context Learning (2023.findings-emnlp)

Challenge:	Large language models (LLMs) face challenges in maintaining accuracy due to the dynamic nature of world knowledge.
Approach:	They propose to use a benchmark dataset to investigate the effects of model edits on model safety metrics and guardrails.
Outcome:	The proposed dataset sheds light on how the edits, impact the model’s safety metrics and guardrails.

Copied to clipboard

Zhuocheng Gong, Jiahao Liu, Qifan Wang, Jingang Wang, Xunliang Cai, Dongyan Zhao, Rui Yan

Neural Incompatibility: The Unbridgeable Gap of Cross-Scale Parametric Knowledge Transfer in Large Language Models (2025.acl-long)

Challenge:	In-context learning (ICL) is an emerging capability of large autoregressive language models where a few demonstrations are appended to the input to enhance the model’s understanding of downstream NLP tasks without directly adjusting the model parameters.
Approach:	They propose a method where a few demonstrations are appended to the input to enhance the model's understanding of downstream NLP tasks without directly adjusting the model parameters.
Outcome:	The proposed method significantly improves the input-label mapping in ICL demonstrations.

Copied to clipboard

Yuqiao Tan, Shizhu He, Kang Liu, Jun Zhao

Unveiling the Power of Argument Arrangement in Online Persuasive Discussions (2023.findings-emnlp)

Challenge:	Large Language Models (LLMs) have a transparent brain with accessible parameters that encode extensive knowledge, which can be analyzed, located and transferred.
Approach:	They propose a new paradigm that aligns parametric spaces of LLMs using several training steps without following training.
Outcome:	The proposed model aligns parametric spaces across scales using only training steps without following training.

Copied to clipboard

Nailia Mirzakhmedova, Johannes Kiesel, Khalid Al-Khatib, Benno Stein

Challenge:	a recent study shows that the CMV is the best time period in human history for the vast majority of people.
Approach:	They extend a semantic argumentation unit type model by clustering type sequences into different argument arrangement patterns and representing discussions as sequences of these patterns.
Outcome:	The proposed model outperforms existing classifiers on the change my view forum discussion data.

Diagnosing Failures in Large Language Models’ Answers: Integrating Error Attribution into Evaluation Framework (2025.findings-acl)

Copied to clipboard

Zishan Xu, Shuyi Xie, Qingsong Lv, Shupei Xiao, Linlin Song, Sui Wenjuan, Fan Lin

Step-by-Step Reasoning to Solve Grid Puzzles: Where do LLMs Falter? (2024.emnlp-main)

Challenge:	Existing evaluation models lack error attribution capability due to their proprietary nature.
Approach:	They propose a misattribution framework with 6 primary and 15 secondary categories to facilitate in-depth analysis.
Outcome:	The proposed framework is based on a dataset specifically designed for error attribution, along with the corresponding scores and feedback.

Copied to clipboard

Nemika Tyagi, Mihir Parmar, Mohith Kulkarni, Aswin Rrv, Nisarg Patel, Mutsumi Nakamura, Arindam Mitra, Chitta Baral

TUMLU: A Unified and Native Language Understanding Benchmark for Turkic Languages (2025.acl-long)

Challenge:	Existing studies evaluate only the final predicted answer of a puzzle, without providing any finer metrics to evaluate them.
Approach:	They propose to use a grid-based evaluation dataset to evaluate LLMs' reasoning abilities and a new error taxonomy to evaluate their reasoning chains.
Outcome:	The proposed model outperforms existing prompting methods on a wide range of natural language understanding tasks previously thought to be exclusive to humans.

Copied to clipboard

Jafar Isbarov, Arofat Akhundjanova, Mammad Hajili, Kavsar Huseynova, Dmitry Gaynullin, Anar Rzayev, Osman Tursun, Aizirek Turdubaeva, Ilshat Saetov, Rinat Kharisov, Saule Belginova, Ariana Kenbayeva, Amina Alisheva, Abdullatif Köksal, Samir Rustamov, Duygu Ataman

LLMs as Planning Formalizers: A Survey for Leveraging Large Language Models to Construct Automated Planning Models (2025.findings-acl)

Challenge:	preparing native language MMLU benchmarks is costly and limits representativeness of evaluation datasets.
Approach:	They propose to use a Turkic language MMLU benchmark to assess massive multitask language understanding capabilities.
Outcome:	The proposed benchmarks are based on a Turkic language morphosyntactic and cultural benchmark . the benchmarks evaluate a diverse range of open and proprietary multilingual large language models .

Copied to clipboard

Marcus Tantakoun, Christian Muise, Xiaodan Zhu

DraDDP: A Multimodal Multi-Party Dialogue Discourse Parsing Dataset (2026.findings-acl)

Challenge:	Large Language Models excel in various natural language tasks but struggle with long-horizon planning problems requiring structured reasoning.
Approach:	They propose to integrate large language models into AP and NLP planning frameworks by reviewing current research and identifying critical challenges and future directions.
Outcome:	The proposed frameworks are used to support reliable off-the-shelf AP planners.

Copied to clipboard

Shannan Liu, Peifeng Li, Yaxin Fan, Qiaoming Zhu

SACL: Understanding and Combating Textual Bias in Code Retrieval with Semantic-Augmented Reranking and Localization (2025.findings-emnlp)

Challenge:	Existing studies on multi-party dialogue discourse parsing focus on textual modality and two-party dialog . et al., 2016) focused on text-based discourse parses, ignoring the complexity and richness of multimodal interactions in real-world scenarios.
Approach:	They construct the first publicly available English multimodal dataset for multi-party dialogue discourse parsing based on American TV dramas.
Outcome:	The proposed dataset contains 495 dialogue segments with 6,374 utterances and 9.1 hours of parallel video content, covering rich multi-party interaction scenarios.

Copied to clipboard

Dhruv Gupta, Gayathri Ganesh Lakshmy, Yiqing Xie

Think Smart, Not Hard: Difficulty Adaptive Reasoning for Large Audio Language Models (2026.findings-acl)

Challenge:	a recent study shows that code retrievers exhibit a strong bias towards well-documented code .
Approach:	They propose a framework that augments textual information with semantic information to mask specific features while preserving code functionality.
Outcome:	The proposed framework enhances textual information and reduces bias by augmenting code or structural knowledge with semantic information.

Copied to clipboard

Zhichao Sheng, Shilin Zhou, Chen Gong, Zhenghua Li

GSM-Noise: Exploring and Enhancing Large Language Models’ Reasoning under Noisy Inputs (2026.findings-acl)

Challenge:	Existing methods to determine whether to perform reasoning lack fine-grained mechanisms to adapt reasoning length to problem complexity.
Approach:	They propose a difficulty-adaptive reasoning method that dynamically links reasoning length to the model’s perceived problem difficulty.
Outcome:	The proposed method reduces average reasoning length by 50%, achieving higher efficiency without sacrificing accuracy.

Copied to clipboard

Zhengxin Zhang, Chengyu Huang, Xufu Liu, Dan Zhao, Jinyan Su, Claire Cardie

When Personalization Tricks Detectors: The Feature-Inversion Trap in Machine-Generated Text Detection (2026.acl-long)

Challenge:	Large language models struggle when dealing with complex, ill-formed, or noisy inputs . open-source models are less robust, while closed-source ones are more robust .
Approach:	They propose to use GSM-Noise to refine inputs before engaging in in-depth analysis to improve LLM robustness under noisy conditions.
Outcome:	The proposed model can achieve consistent performance gains under noisy conditions with prompt engineering, supervised finetuning, and reinforcement learning.

Copied to clipboard

Lang Gao, Xuhui Li, Chenxi Wang, Mingzhe Li, Wei Liu, Zirui Song, Jinghui Zhang, Rui Yan, Preslav Nakov, Xiuying Chen